## Changes
Add a `MigrationSequencer` class to sequence the migration steps for
jobs.

The PR includes the following resources in its sequence:
- Jobs
- Job tasks
- Job tasks dependencies
- Job clusters
- Cluster

Other elements part of the sequence are added later

### Linked issues
Progresses #1415 
Supersedes #2980 

### Tests
- [x] added unit tests
- [x] added integration tests

---------

Co-authored-by: Eric Vergnaud <[email protected]>
Co-authored-by: Cor Zuurmond <[email protected]>

* Added `MigrationSequencer` for jobs ([#3008](#3008)). In this commit, a `MigrationSequencer` class has been added to manage the migration sequence for various resources including jobs, job tasks, job task dependencies, job clusters, and clusters. The class builds a graph of dependencies and analyzes it to generate the migration sequence, which is returned as an iterable of `MigrationStep` objects. These objects contain information about the object type, ID, name, owner, required step IDs, and step number. The commit also includes new unit and integration tests to ensure the functionality is working correctly. The migration sequence is used in tests for assessing the sequencing feature, and it handles tasks that reference existing or non-existing clusters or job clusters, and new cluster definitions. This change is linked to issue [#1415](#1415) and supersedes issue [#2980](#2980). Additionally, the commit removes some unnecessary imports and fixtures from a test file.
* Added `phik` to known list ([#3198](#3198)). In this release, we have added `phik` to the known list in the provided JSON file. This change addresses part of issue [#1931](#1931), as outlined in the linked issues. The `phik` key has been added with an empty list as its value, consistent with the structure of other keys in the JSON file. It is important to note that no existing functionality has been altered and no new methods have been introduced in this commit. The scope of the change is confined to updating the known list in the JSON file by adding the `phik` key.
* Added `pmdarima` to known list ([#3199](#3199)). In this release, we are excited to announce the addition of support for the `pmdarima` library, an open-source Python library for automatic seasonal decomposition of time series. With this commit, we have added `pmdarima` to our known list of libraries, providing our users with access to its various methods and functions for data preprocessing, model selection, and visualization. The library is particularly useful for fitting ARIMA models and testing for seasonality. By integrating `pmdarima`, users can now perform time series analysis and forecasting with greater ease and efficiency. This change partly resolves issue [#1931](#1931) and underscores our commitment to providing our users with access to the latest and most innovative open-source libraries available.
* Added `preshed` to known list ([#3220](#3220)). A new library, "preshed," has been added to our project's supported libraries, enhancing compatibility and enabling efficient utilization of its capabilities. Developed using Cython, `preshed` is a Python interface to Intel(R) MKL's sparse BLAS, sparse solvers, and sparse linear algebra routines. With the inclusion of two modules, `preshed` and "preshed.about," this addition partially resolves issue [#1931](#1931), improving the project's overall performance and reliability in sparse linear algebra tasks. Software engineers can now leverage the `preshed` library's features and optimized routines for their projects, reducing development time and increasing efficiency.
* Added `py-cpuinfo` to known list ([#3221](#3221)). In this release, we have added support for the `py-cpuinfo` library to our project, enabling the use of the `cpuinfo` functionality that it provides. With this addition, developers can now access detailed information about the CPU, such as the number of cores, current frequency, and vendor, which can be useful for performance tuning and optimization. This change partially resolves issue [#1931](#1931) and does not affect any existing functionality or add new methods to the codebase. We believe that this improvement will enhance the capabilities of our project and enable more efficient use of CPU resources.
* Cater for empty python cells ([#3212](#3212)). In this release, we have resolved an issue where certain notebook cells in the dependency builder were causing crashes. Specifically, empty or comment-only cells were identified as the source of the problem. To address this, we have implemented a check to account for these cases, ensuring that an empty tree is stored in the `_python_trees` dictionary if the input cell does not produce a valid tree. This change helps prevent crashes in the dependency builder caused by empty or comment-only cells. Furthermore, we have added a test to verify the fix on a failed repository. If a cell does not produce a tree, the `_load_children_from_tree` method will not be executed for that cell, skipping the loading of any children trees. This enhancement improves the overall stability and reliability of the library by preventing crashes caused by invalid input.
* Create `TODO` issues every nightly run ([#3196](#3196)). A commit has been made to update the `acceptance` repository version in the `acceptance.yml` GitHub workflow from `acceptance/v0.4.0` to `acceptance/v0.4.2`, which affects the integration tests. The `Run nightly tests` step in the GitHub repository's workflow has also been updated to use a newer version of the `databrickslabs/sandbox/acceptance` action, from `v0.3.1` to `v0.4.2`. Software engineers should verify that the new version of the `acceptance` repository contains all necessary updates and fixes, and that the integration tests continue to function as expected. Additionally, testing the updated action is important to ensure that the nightly tests run successfully with up-to-date code and can catch potential issues.
* Fixed Integration test failure of migration_tables ([#3108](#3108)). This release includes a fix for two integration tests (`test_migrate_managed_table_to_external_table_without_conversion` and `test_migrate_managed_table_to_external_table_with_clone`) related to Hive Metastore table migration, addressing issues [#3054](#3054) and [#3055](#3055). Previously skipped due to underlying problems, these tests have now been unskipped, enhancing the migration feature's test coverage. No changes have been made to the existing functionality, as the focus is solely on including the previously skipped tests in the testing suite. The changes involve removing `@pytest.mark.skip` markers from the test functions, ensuring they run and provide a more comprehensive test coverage for the Hive Metastore migration feature. In addition, this release includes an update to DirectFsAccess integration tests, addressing issues related to the removal of DFSA collectors and ensuring proper handling of different file types, with no modifications made to other parts of the codebase.
* Replace MockInstallation with MockPathLookup for testing fixtures ([#3215](#3215)). In this release, we have updated the testing fixtures in our unit tests by replacing the MockInstallation class with MockPathLookup. Specifically, we have modified the _load_sources function to use MockPathLookup instead of MockInstallation for loading sources. This change not only enhances the testing capabilities of the module but also introduces a new logger, logger, for more precise logging within the module. Additionally, we have updated the _load_sources function calls in the test_notebook.py file to pass the file path directly instead of a SourceContainer object. This modification allows for more flexible and straightforward testing of file-related functionality, thereby fixing issue [#3115](#3115).
* Updated sqlglot requirement from <25.29,>=25.5.0 to >=25.5.0,<25.30 ([#3224](#3224)). The open-source library `sqlglot` has been updated to version 25.29.0 with this release, incorporating several breaking changes, new features, and bug fixes. The breaking changes include transpiling `ANY` to `EXISTS`, supporting the `MEDIAN()` function, wrapping values in `NOT value IS ...`, and parsing information schema views into a single identifier. New features include support for the `JSONB_EXISTS` function in PostgreSQL, transpiling `ANY` to `EXISTS` in Spark, transpiling Snowflake's `TIMESTAMP()` function, and adding support for hexadecimal literals in Teradata. Bug fixes include handling a Move edge case in the semantic differ, adding a `NULL` filter on `ARRAY_AGG` only for columns, improving parsing of `WITH FILL ... INTERPOLATE` in Clickhouse, generating `LOG(...)` for `exp.Ln` in TSQL, and optionally parsing a Stream expression. The full changelog can be found in the pull request, which also includes a list of the commits included in this release.
* Use acceptance/v0.4.0 ([#3192](#3192)). A change has been made to the GitHub Actions workflow file for acceptance tests, updating the version of the `databrickslabs/sandbox/acceptance` runner to `acceptance/v0.4.0` and granting write permissions for the `issues` field in the `permissions` section. These updates will allow for the use of the latest version of the acceptance tests and provide the necessary permissions to interact with issues. A `TODO` comment has been added to indicate that the new version of the acceptance tests needs to be updated elsewhere in the codebase. This change will ensure that the acceptance tests are up-to-date and functioning properly.
* Warn about errors instead to avoid job task failure ([#3219](#3219)). In this change, the `refresh_report` method in `jobs.py` has been updated to log warnings instead of raising errors when certain problems are encountered during its execution. Previously, if there were any errors during the linting process, a `ManyError` exception was raised, causing the job task to fail. Now, errors are logged as warnings, allowing the job task to continue running successfully. This resolves issue [#3214](#3214) and ensures that the job task will not fail due to linting errors, allowing users to be aware of any issues that occurred during the linting process while still completing the job task successfully. The updated method checks for errors during the linting process, adds them to a list, and constructs a string of error messages if there are any. This string of error messages is then logged as a warning using the `logger.warning` function, allowing the method to continue executing and the job task to complete successfully.
* [DOC] Add dashboard section ([#3222](#3222)). In this release, we have added a new dashboard section to the project documentation, which provides visualizations of UCX's outcomes to help users better understand and manage their UCX environment. The new section includes a table listing the available dashboards, including the Azure service principals dashboard. This dashboard displays information about Azure service principals discovered by UCX in configurations from various sources such as clusters, cluster policies, job clusters, pipelines, and warehouses. Each dashboard has text widgets that offer detailed information about the contents and are designed to help users understand UCX's results and progress in a more visual and interactive way. The Azure service principals dashboard specifically offers users valuable insights into their Azure service principals within the UCX environment.
* [DOC] README.md rewrite ([#3211](#3211)). The Databricks Labs UCX package offers a suite of tools for migrating data objects from the Hive metastore to Unity Catalog (UC), encompassing a comprehensive table migration process. This process consists of table mapping, data access setup, creating new UC resources, and migrating Hive metastore data objects. Table mapping is achieved using a table mapping file that defaults to mapping all tables/views to UC tables while preserving the original schema and names, but can be customized as needed. Data access setup involves creating and modifying cloud principals and credentials for UC data. New UC resources are created without affecting existing Hive metastore resources, and users can choose from various strategies for migrating tables based on their format and location. Additionally, the package provides installation resources, including a README notebook, a DEBUG notebook, debug logs, and installation configuration, as well as utility commands for viewing and repairing workflows. The migration process also includes an assessment workflow, group migration workflow, data reconciliation, and code migration commands.
* [chore] Added tests to verify linter not being stuck in the infinite loop ([#3225](#3225)). In this release, we have added new functional tests to ensure that the linter does not get stuck in an infinite loop, addressing a bug that was fixed in version 0.46.0 related to the default format change from Parquet to Delta in Databricks Runtime 8.0 and a SQL parse error. These tests involve creating data frames, writing them to tables, and reading from those tables, using PySpark's SQL functions and a system information schema table to demonstrate the corrected behavior. The tests also include SQL queries that select columns from a system information schema table with a specified limit, using a withColumn() method to add a new column to a data frame based on a condition. These new tests provide assurance that the linter will not get stuck in an infinite loop and that SQL queries with table parameters are supported.
* [internal] Temporarily disable integration tests due to ES-1302145 ([#3226](#3226)). In this release, the integration tests for moving tables, views, and aliasing tables have been temporarily disabled due to issue ES-1302145. The `test_move_tables`, `test_move_views`, and `test_alias_tables` functions were previously decorated with `@retried` to handle potential `NotFound` exceptions and had a timeout of 2 minutes, but are now marked with `@pytest.mark.skip("ES-1302145")`. Once the issue is resolved, the `@pytest.mark.skip` decorator should be removed to re-enable the tests. The remaining code in the file, including the `test_move_tables_no_from_schema`, `test_move_tables_no_to_schema`, and `test_move_views_no_from_schema` functions, is unchanged and still functional.
* use a path instance for MISSING_SOURCE_PATH and add test ([#3217](#3217)). In this release, the handling of MISSING_SOURCE_PATH has been improved by replacing the string representation with a Path instance using Pathlib, which simplifies checks for missing source paths and enables the addition of a new test for the DependencyProblem class. This test verifies the behavior of the newly introduced method, is_path_missing(), in the DependencyProblem class for determining if a given problem is caused by a missing path. Co-authored by Eric Vergnaud, these changes not only improve the handling and testing of missing paths but also contribute to enhancing the source code analysis functionality of the databricks/labs/ucx project.

Dependency updates:

 * Updated sqlglot requirement from <25.29,>=25.5.0 to >=25.5.0,<25.30 ([#3224](#3224)).

* Added `MigrationSequencer` for jobs
([#3008](#3008)). In this
commit, a `MigrationSequencer` class has been added to manage the
migration sequence for various resources including jobs, job tasks, job
task dependencies, job clusters, and clusters. The class builds a graph
of dependencies and analyzes it to generate the migration sequence,
which is returned as an iterable of `MigrationStep` objects. These
objects contain information about the object type, ID, name, owner,
required step IDs, and step number. The commit also includes new unit
and integration tests to ensure the functionality is working correctly.
The migration sequence is used in tests for assessing the sequencing
feature, and it handles tasks that reference existing or non-existing
clusters or job clusters, and new cluster definitions. This change is
linked to issue
[#1415](#1415) and
supersedes issue
[#2980](#2980).
Additionally, the commit removes some unnecessary imports and fixtures
from a test file.
* Added `phik` to known list
([#3198](#3198)). In this
release, we have added `phik` to the known list in the provided JSON
file. This change addresses part of issue
[#1931](#1931), as outlined
in the linked issues. The `phik` key has been added with an empty list
as its value, consistent with the structure of other keys in the JSON
file. It is important to note that no existing functionality has been
altered and no new methods have been introduced in this commit. The
scope of the change is confined to updating the known list in the JSON
file by adding the `phik` key.
* Added `pmdarima` to known list
([#3199](#3199)). In this
release, we are excited to announce the addition of support for the
`pmdarima` library, an open-source Python library for automatic seasonal
decomposition of time series. With this commit, we have added `pmdarima`
to our known list of libraries, providing our users with access to its
various methods and functions for data preprocessing, model selection,
and visualization. The library is particularly useful for fitting ARIMA
models and testing for seasonality. By integrating `pmdarima`, users can
now perform time series analysis and forecasting with greater ease and
efficiency. This change partly resolves issue
[#1931](#1931) and
underscores our commitment to providing our users with access to the
latest and most innovative open-source libraries available.
* Added `preshed` to known list
([#3220](#3220)). A new
library, "preshed," has been added to our project's supported libraries,
enhancing compatibility and enabling efficient utilization of its
capabilities. Developed using Cython, `preshed` is a Python interface to
Intel(R) MKL's sparse BLAS, sparse solvers, and sparse linear algebra
routines. With the inclusion of two modules, `preshed` and
"preshed.about," this addition partially resolves issue
[#1931](#1931), improving
the project's overall performance and reliability in sparse linear
algebra tasks. Software engineers can now leverage the `preshed`
library's features and optimized routines for their projects, reducing
development time and increasing efficiency.
* Added `py-cpuinfo` to known list
([#3221](#3221)). In this
release, we have added support for the `py-cpuinfo` library to our
project, enabling the use of the `cpuinfo` functionality that it
provides. With this addition, developers can now access detailed
information about the CPU, such as the number of cores, current
frequency, and vendor, which can be useful for performance tuning and
optimization. This change partially resolves issue
[#1931](#1931) and does not
affect any existing functionality or add new methods to the codebase. We
believe that this improvement will enhance the capabilities of our
project and enable more efficient use of CPU resources.
* Cater for empty python cells
([#3212](#3212)). In this
release, we have resolved an issue where certain notebook cells in the
dependency builder were causing crashes. Specifically, empty or
comment-only cells were identified as the source of the problem. To
address this, we have implemented a check to account for these cases,
ensuring that an empty tree is stored in the `_python_trees` dictionary
if the input cell does not produce a valid tree. This change helps
prevent crashes in the dependency builder caused by empty or
comment-only cells. Furthermore, we have added a test to verify the fix
on a failed repository. If a cell does not produce a tree, the
`_load_children_from_tree` method will not be executed for that cell,
skipping the loading of any children trees. This enhancement improves
the overall stability and reliability of the library by preventing
crashes caused by invalid input.
* Create `TODO` issues every nightly run
([#3196](#3196)). A commit
has been made to update the `acceptance` repository version in the
`acceptance.yml` GitHub workflow from `acceptance/v0.4.0` to
`acceptance/v0.4.2`, which affects the integration tests. The `Run
nightly tests` step in the GitHub repository's workflow has also been
updated to use a newer version of the
`databrickslabs/sandbox/acceptance` action, from `v0.3.1` to `v0.4.2`.
Software engineers should verify that the new version of the
`acceptance` repository contains all necessary updates and fixes, and
that the integration tests continue to function as expected.
Additionally, testing the updated action is important to ensure that the
nightly tests run successfully with up-to-date code and can catch
potential issues.
* Fixed Integration test failure of migration_tables
([#3108](#3108)). This
release includes a fix for two integration tests
(`test_migrate_managed_table_to_external_table_without_conversion` and
`test_migrate_managed_table_to_external_table_with_clone`) related to
Hive Metastore table migration, addressing issues
[#3054](#3054) and
[#3055](#3055). Previously
skipped due to underlying problems, these tests have now been unskipped,
enhancing the migration feature's test coverage. No changes have been
made to the existing functionality, as the focus is solely on including
the previously skipped tests in the testing suite. The changes involve
removing `@pytest.mark.skip` markers from the test functions, ensuring
they run and provide a more comprehensive test coverage for the Hive
Metastore migration feature. In addition, this release includes an
update to DirectFsAccess integration tests, addressing issues related to
the removal of DFSA collectors and ensuring proper handling of different
file types, with no modifications made to other parts of the codebase.
* Replace MockInstallation with MockPathLookup for testing fixtures
([#3215](#3215)). In this
release, we have updated the testing fixtures in our unit tests by
replacing the MockInstallation class with MockPathLookup. Specifically,
we have modified the _load_sources function to use MockPathLookup
instead of MockInstallation for loading sources. This change not only
enhances the testing capabilities of the module but also introduces a
new logger, logger, for more precise logging within the module.
Additionally, we have updated the _load_sources function calls in the
test_notebook.py file to pass the file path directly instead of a
SourceContainer object. This modification allows for more flexible and
straightforward testing of file-related functionality, thereby fixing
issue [#3115](#3115).
* Updated sqlglot requirement from <25.29,>=25.5.0 to >=25.5.0,<25.30
([#3224](#3224)). The
open-source library `sqlglot` has been updated to version 25.29.0 with
this release, incorporating several breaking changes, new features, and
bug fixes. The breaking changes include transpiling `ANY` to `EXISTS`,
supporting the `MEDIAN()` function, wrapping values in `NOT value IS
...`, and parsing information schema views into a single identifier. New
features include support for the `JSONB_EXISTS` function in PostgreSQL,
transpiling `ANY` to `EXISTS` in Spark, transpiling Snowflake's
`TIMESTAMP()` function, and adding support for hexadecimal literals in
Teradata. Bug fixes include handling a Move edge case in the semantic
differ, adding a `NULL` filter on `ARRAY_AGG` only for columns,
improving parsing of `WITH FILL ... INTERPOLATE` in Clickhouse,
generating `LOG(...)` for `exp.Ln` in TSQL, and optionally parsing a
Stream expression. The full changelog can be found in the pull request,
which also includes a list of the commits included in this release.
* Use acceptance/v0.4.0
([#3192](#3192)). A change
has been made to the GitHub Actions workflow file for acceptance tests,
updating the version of the `databrickslabs/sandbox/acceptance` runner
to `acceptance/v0.4.0` and granting write permissions for the `issues`
field in the `permissions` section. These updates will allow for the use
of the latest version of the acceptance tests and provide the necessary
permissions to interact with issues. A `TODO` comment has been added to
indicate that the new version of the acceptance tests needs to be
updated elsewhere in the codebase. This change will ensure that the
acceptance tests are up-to-date and functioning properly.
* Warn about errors instead to avoid job task failure
([#3219](#3219)). In this
change, the `refresh_report` method in `jobs.py` has been updated to log
warnings instead of raising errors when certain problems are encountered
during its execution. Previously, if there were any errors during the
linting process, a `ManyError` exception was raised, causing the job
task to fail. Now, errors are logged as warnings, allowing the job task
to continue running successfully. This resolves issue
[#3214](#3214) and ensures
that the job task will not fail due to linting errors, allowing users to
be aware of any issues that occurred during the linting process while
still completing the job task successfully. The updated method checks
for errors during the linting process, adds them to a list, and
constructs a string of error messages if there are any. This string of
error messages is then logged as a warning using the `logger.warning`
function, allowing the method to continue executing and the job task to
complete successfully.
* [DOC] Add dashboard section
([#3222](#3222)). In this
release, we have added a new dashboard section to the project
documentation, which provides visualizations of UCX's outcomes to help
users better understand and manage their UCX environment. The new
section includes a table listing the available dashboards, including the
Azure service principals dashboard. This dashboard displays information
about Azure service principals discovered by UCX in configurations from
various sources such as clusters, cluster policies, job clusters,
pipelines, and warehouses. Each dashboard has text widgets that offer
detailed information about the contents and are designed to help users
understand UCX's results and progress in a more visual and interactive
way. The Azure service principals dashboard specifically offers users
valuable insights into their Azure service principals within the UCX
environment.
* [DOC] README.md rewrite
([#3211](#3211)). The
Databricks Labs UCX package offers a suite of tools for migrating data
objects from the Hive metastore to Unity Catalog (UC), encompassing a
comprehensive table migration process. This process consists of table
mapping, data access setup, creating new UC resources, and migrating
Hive metastore data objects. Table mapping is achieved using a table
mapping file that defaults to mapping all tables/views to UC tables
while preserving the original schema and names, but can be customized as
needed. Data access setup involves creating and modifying cloud
principals and credentials for UC data. New UC resources are created
without affecting existing Hive metastore resources, and users can
choose from various strategies for migrating tables based on their
format and location. Additionally, the package provides installation
resources, including a README notebook, a DEBUG notebook, debug logs,
and installation configuration, as well as utility commands for viewing
and repairing workflows. The migration process also includes an
assessment workflow, group migration workflow, data reconciliation, and
code migration commands.
* [chore] Added tests to verify linter not being stuck in the infinite
loop ([#3225](#3225)). In
this release, we have added new functional tests to ensure that the
linter does not get stuck in an infinite loop, addressing a bug that was
fixed in version 0.46.0 related to the default format change from
Parquet to Delta in Databricks Runtime 8.0 and a SQL parse error. These
tests involve creating data frames, writing them to tables, and reading
from those tables, using PySpark's SQL functions and a system
information schema table to demonstrate the corrected behavior. The
tests also include SQL queries that select columns from a system
information schema table with a specified limit, using a withColumn()
method to add a new column to a data frame based on a condition. These
new tests provide assurance that the linter will not get stuck in an
infinite loop and that SQL queries with table parameters are supported.
* [internal] Temporarily disable integration tests due to ES-1302145
([#3226](#3226)). In this
release, the integration tests for moving tables, views, and aliasing
tables have been temporarily disabled due to issue ES-1302145. The
`test_move_tables`, `test_move_views`, and `test_alias_tables` functions
were previously decorated with `@retried` to handle potential `NotFound`
exceptions and had a timeout of 2 minutes, but are now marked with
`@pytest.mark.skip("ES-1302145")`. Once the issue is resolved, the
`@pytest.mark.skip` decorator should be removed to re-enable the tests.
The remaining code in the file, including the
`test_move_tables_no_from_schema`, `test_move_tables_no_to_schema`, and
`test_move_views_no_from_schema` functions, is unchanged and still
functional.
* use a path instance for MISSING_SOURCE_PATH and add test
([#3217](#3217)). In this
release, the handling of MISSING_SOURCE_PATH has been improved by
replacing the string representation with a Path instance using Pathlib,
which simplifies checks for missing source paths and enables the
addition of a new test for the DependencyProblem class. This test
verifies the behavior of the newly introduced method, is_path_missing(),
in the DependencyProblem class for determining if a given problem is
caused by a missing path. Co-authored by Eric Vergnaud, these changes
not only improve the handling and testing of missing paths but also
contribute to enhancing the source code analysis functionality of the
databricks/labs/ucx project.

Dependency updates:

* Updated sqlglot requirement from <25.29,>=25.5.0 to >=25.5.0,<25.30
([#3224](#3224)).